Complex Document Information Processing: Towards Software and Test Collections

Complex document information processing (CDIP) is the analysis of combinations of handwritten notes, diagrams, and graphics in addition to printed and formatted text. Rapid analysis of collections of complex documents is increasingly critical to intelligence appraisals of hostile forces and other entities of interest. Such collections may be seized by military or law enforcement (recently in Iraq, Afghanistan, and elsewhere), or turned over by cooperating parties (e.g. defectors). This material must be scanned or otherwise input, understood, and exploited all in a short period of time if it is to be of maximum value. Such collections may contain many types of complex documents and data, and ideally should be interpreted in the context of previously obtained collections and knowledge. Unfortunately, most current tools are specialized to one kind of data (text content or style, handwriting, graphics, database records, etc.) and sometimes only one genre of document. Extracting intelligence from such collections therefore relies heavily on manual partitioning of data, coordination of analysis tools, and collation of results. Besides being slow, expensive, and demanding on the time of trained analysts, the current approach has two fundamental problems. First, the speed and accuracy of analyzing one data type can often be improved if partial analyses from other data types were available, but a collection of standalone tools does not support this. Second, manual collation, cross-checking, and consistency checking is slow and error-prone when dealing with large numbers of documents, further reducing the value of obtained information. The operational problem, therefore, is the dearth of unified tools for CDIP in both commercial and research spheres. That in itself, however, is symptomatic of a second problem: the lack of standard test collections for CDIP research. Test collections of documents (and manual judgments/annotations) in text retrieval and categorization, handwriting recognition, and data mining have sparked intense research and commercial development. Conversely, CDIP research and development is stifled by the lack of a test collection of complex documents. Creating a CDIP test collection is thus an essential component of any serious CDIP effort. II. BUILDING A CDIP SYSTEM